Feat: multiple endpoints using a list of LitServer #276

bhimrazy · 2024-09-12T08:11:54Z

Before submitting

Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure to update the docs?
Did you write any new necessary tests?

⚠️ How does this PR impact the user? ⚠️

As a user, I want to host multiple endpoints for different purposes, such as serving an embedding API, prediction API, etc., on the same server while maintaining LitServer features.

What does this PR do?

Fixes #271.

This PR introduces a feature that allows running multiple LitServer instances in a combined form, as discussed in issue Is it possible to support multiple endpoints for one server? #271.

Usage

# server.py
from litserve.server import LitServer, run_all
from litserve.test_examples import SimpleLitAPI


class SimpleLitAPI1(SimpleLitAPI):
    def setup(self, device):
        self.model = lambda x: x**1


class SimpleLitAPI2(SimpleLitAPI):
    def setup(self, device):
        self.model = lambda x: x**2


class SimpleLitAPI3(SimpleLitAPI):
    def setup(self, device):
        self.model = lambda x: x**3


class SimpleLitAPI4(SimpleLitAPI):
    def setup(self, device):
        self.model = lambda x: x**4


if __name__ == "__main__":
    server1 = LitServer(SimpleLitAPI1(), api_path="/predict-1")
    server2 = LitServer(SimpleLitAPI2(), api_path="/predict-2")
    server3 = LitServer(SimpleLitAPI3(), api_path="/predict-3")
    server4 = LitServer(SimpleLitAPI4(), api_path="/predict-4")
    run_all([server1, server2, server3, server4], port=8000)

# client.py
import requests

for i in range(1, 5):
    resp = requests.post(f"http://127.0.0.1:8000/predict-{i}", json={"input": 4.0}, headers=None)
    assert resp.status_code == 200, f"Expected response to be 200 but got {resp.status_code}"
    assert resp.json() == {"output": 4.0**i}, f"Expected response to be {4.0**i} but got {resp.json()}"

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

for more information, see https://pre-commit.ci

codecov · 2024-09-12T08:16:09Z

Codecov Report

Attention: Patch coverage is 98.43750% with 1 line in your changes missing coverage. Please review.

Project coverage is 95%. Comparing base (f475369) to head (e5db967).

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #276   +/-   ##
===================================
  Coverage    95%    95%           
===================================
  Files        14     14           
  Lines      1082   1143   +61     
===================================
+ Hits       1025   1085   +60     
- Misses       57     58    +1

for more information, see https://pre-commit.ci

aniketmaurya

Awesome @bhimrazy!! We might have to look into mounting FastAPI app instances like here so that the endpoints are visible from all the uvicorn servers.

src/litserve/server.py

for more information, see https://pre-commit.ci

…nto feat/multi-endpoints

…rvers list

…er.py

for more information, see https://pre-commit.ci

src/litserve/server.py

…stency

…stency and minor cleanup

src/litserve/server.py

williamFalcon

ok, let's pause this for now while we validate with customers what is a real use case...
rushing into something will only complicate the codebase, without no real user proof of exactly what problems this solves for them...

this feels very "theoretical" at this stage... i want to root this with user feedback first @lantiga @aniketmaurya

aceliuchanghong · 2024-09-25T14:12:02Z

still can't use this function

VikramxD · 2024-10-02T14:46:12Z

In the meantime , this gets merged , is there any other way to run multiple endpoints in one main server ?

williamFalcon · 2024-10-02T15:36:39Z

this is paused and not scheduled to be merged until we have a very clear usecase.

so, the best way to unblock this is to share code of what you are trying to do and why you wouldn’t just run two separate servers on the same machine?

@VikramxD

bhimrazy added 2 commits September 12, 2024 02:04

Adds initial function definition

1ec0782

adds run all method

88b6b2f

bhimrazy requested review from lantiga, aniketmaurya, awaelchli and Andrei-Aksionov as code owners September 12, 2024 08:11

bhimrazy marked this pull request as draft September 12, 2024 08:11

[pre-commit.ci] auto fixes from pre-commit.com hooks

30d2c35

for more information, see https://pre-commit.ci

bhimrazy and others added 2 commits September 12, 2024 14:06

adding a test for multi endpoint servers

266b271

[pre-commit.ci] auto fixes from pre-commit.com hooks

3cf17a0

for more information, see https://pre-commit.ci

aniketmaurya reviewed Sep 12, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

bhimrazy and others added 16 commits September 12, 2024 14:35

fixes multi client generation

f4b4715

[pre-commit.ci] auto fixes from pre-commit.com hooks

ab90959

for more information, see https://pre-commit.ci

testing mounting of apps

e3ee2b9

mounted on root

e1865df

Merge branch 'main' into feat/multi-endpoints

faea18a

Refactor run_all function to support multiple LitServers

49865f2

[pre-commit.ci] auto fixes from pre-commit.com hooks

368f91c

for more information, see https://pre-commit.ci

adds test for the manage_lifespan

885ab06

ref: format imports

7ba3ab7

Merge branch 'feat/multi-endpoints' of github.com:bhimrazy/LitServe i…

fe8f82c

…nto feat/multi-endpoints

removed the extra test file

d7cb562

adds simple server with multi endpoints

d502af4

adds e2e test for multi endpoints

17a37cc

Refactor run_all function to remove unnecessary check for empty litse…

452a509

…rvers list

adds test to runnall litservers

391cefd

Refactor test_e2e_with_multi_endpoints function name

ea0ac77

bhimrazy changed the title ~~[WIP]: Add support for multi-endpoints~~ Feat: Add support for multi-endpoints Sep 15, 2024

Refactor test_e2e_with_multi_endpoints function name

b82733e

bhimrazy changed the title ~~Feat: Add support for multi-endpoints~~ Feat: Add support for running multi-litservers in combined form Sep 17, 2024

bhimrazy changed the title ~~Feat: Add support for running multi-litservers in combined form~~ Feat: Add support for running multiple litservers in combined form Sep 17, 2024

bhimrazy and others added 7 commits September 17, 2024 11:43

Refactor manage_lifespan to multi_server_lifespan

e061c88

Refactor test multi_server_lifespan

1de6935

refactor classnames

0697222

Refactor test_e2e_combined_multiple_litserver to use multiple_litserv…

4ac180d

…er.py

adds default queue id for the main app

3e9dc6e

[pre-commit.ci] auto fixes from pre-commit.com hooks

def24f8

for more information, see https://pre-commit.ci

rm num api server

d19e1ae

bhimrazy commented Sep 17, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

Update server.py

3705505

aniketmaurya reviewed Sep 17, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

fix

fa8dd7d

aniketmaurya changed the title ~~Feat: Add support for running multiple litservers in combined form~~ Feat: multiple endpoints using a list of LitServer Sep 17, 2024

Borda reviewed Sep 17, 2024

View reviewed changes

src/litserve/server.py Show resolved Hide resolved

Borda reviewed Sep 17, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

Borda approved these changes Sep 17, 2024

View reviewed changes

bhimrazy added 4 commits September 18, 2024 00:52

Refactor server.py to use 'servers' instead of 'litservers' for consi…

724d80b

…stency

added more test cases to increase the coverage

2ac6388

updated test

a09f540

Refactor server.py to use 'servers' instead of 'litservers' for consi…

e5db967

…stency and minor cleanup

bhimrazy commented Sep 18, 2024

View reviewed changes

src/litserve/server.py Show resolved Hide resolved

bhimrazy added 2 commits September 18, 2024 15:19

Merge branch 'main' into feat/multi-endpoints

efb838c

update

33e3b12

williamFalcon self-requested a review September 18, 2024 16:33

williamFalcon requested changes Sep 18, 2024

View reviewed changes

bhimrazy marked this pull request as draft September 18, 2024 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: multiple endpoints using a list of LitServer #276

Feat: multiple endpoints using a list of LitServer #276

bhimrazy commented Sep 12, 2024 •

edited

Loading

codecov bot commented Sep 12, 2024 •

edited

Loading

aniketmaurya left a comment

williamFalcon left a comment

aceliuchanghong commented Sep 25, 2024

VikramxD commented Oct 2, 2024

williamFalcon commented Oct 2, 2024

Feat: multiple endpoints using a list of LitServer #276

Are you sure you want to change the base?

Feat: multiple endpoints using a list of LitServer #276

Conversation

bhimrazy commented Sep 12, 2024 • edited Loading

⚠️ How does this PR impact the user? ⚠️

What does this PR do?

PR review

Did you have fun?

codecov bot commented Sep 12, 2024 • edited Loading

Codecov Report

aniketmaurya left a comment

Choose a reason for hiding this comment

williamFalcon left a comment

Choose a reason for hiding this comment

aceliuchanghong commented Sep 25, 2024

VikramxD commented Oct 2, 2024

williamFalcon commented Oct 2, 2024

bhimrazy commented Sep 12, 2024 •

edited

Loading

codecov bot commented Sep 12, 2024 •

edited

Loading